Repository navigation
bench(arrow): merge with a scalar side over run masks, a NULL scalar and decimals - #11527
mightsleep wants to merge 3 commits into
Conversation
|
|
||
| let mut group = | ||
| c.benchmark_group(format!("merge_{ARRAY_LEN}_from_{}", input_generator.name()).as_str()); | ||
| // One merge of 8192 rows takes microseconds, so the default 3 s warm up |
There was a problem hiding this comment.
i wonder if we need to cut down the benches in this suite; similar to what we did for zip
and/or reduce from 8192 to a smaller batch size 🤔
There was a problem hiding this comment.
Yes hello, agreed. Cut it down the same way as #10919: one strings (3..20) type instead of short/long strings and bytes, dropped non_null_scalar_vs_null_scalar, and kept two of the four run masks (runs of 64 at 50 %, runs of 512 at 10 %). 448 -> 168 cases, under the 240 on main. I kept 8192 rows: at smaller batches the fixed per-call cost starts to dominate and the per-row difference this suite is meant to show gets harder to see.
e5abf50 to
853df32
Compare
|
Yes hello, one more change: each case now cycles through 64 masks and arrays, as zip_kernels does since #10919. With a single mask the branch predictor learns its 128 words, and on main's kernel that made random masks look 1.2 to 1.45x faster than with fresh ones (i32 array_vs_non_null_scalar/90pct_true 16.6 vs 23.9 µs). Still 168 cases. Also rebased on main for the CI fix. |
Which issue does this PR close?
Rationale for this change
mergewith one scalar side is what DataFusion runs forCASE WHEN c THEN expr ELSE literal END, and today it extendsMutableArrayDataonce per row for the scalar side, about 5 to 6 ns per row. The existing benchmark covers array against non-null scalar for i32 and the byte types on random masks. Before changing the kernel, this adds the shapes the next PR (primitives and decimals) needs to be measured on:CASE WHEN ... THEN array END)Boolean and Utf8View follow in a small benchmark PR before the PRs that change those paths.
What changes are included in this PR?
create_clustered_array, the same run model as thefilter_bitsbenchmark from bench(arrow-select): filter_bits over batches of random and clustered masks #11271, and four run masks.Baseline on main: i32 with a scalar side 5 to 6 ns per row, a NULL scalar 8.7 ns per row, and a run mask (
runs_of_512_10pct_true) still 43 µs for 8192 rows, as the kernel goes row by row inside runs too.Are these changes tested?
Benchmark only.
cargo bench -p arrow --bench merge_kernels --features test_utilsruns all 448 cases.Are there any user-facing changes?
No.